REFLECTIONS ON RUSSIAN 
ACCIDENT ON AUGUST 17" 
2009 





6,400-MW SAYANO- 
SHUSHENSKAYA 
HYDROELECTRI C POWER PLANT 





Purpose of presentation is to learn from the 
accident and use this experience in the 
insurance engineering surveys. 


The official report was used, as well as 
Russian Internet information 


By Eugenio Kolesnikov — Miami October 12 
2009; revised on October 20. 


www.rudhydro.ru - official site of the owner 
http:// www. 1tv.ru - Russian TV channel 1 
http: //forums.drom.ru/hakasiya - discussion web site 


http: //www.1tv.ru/news/techno/152840 - final report 






Officially the loss report was 
presented on October 3'% 2009 
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Стали известны причины — 
нарми на Саяно-Шушенской 
rə 


Сегодня специальная комиссия 
завершила расследование обстоятельств 
аварии на Саяно-Шушенской ГЭС. 


Названы имена людей, чьи действия или, 
наоборот, бездействие послужили началом 
целой цепочки ошибок, повлекшей за 
собой гибель сотрудников станции и 
разрушения на ГЭС. Эксперты отмечают 
совпадение ряда причин, которые в итоге 
привели к трагедии: ошибки при 
проектировании, неправильные 
эксплуатация и ремонт. 
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General Information 


6'^ world largest power plant. 

Useful life of the units is 30 years by design 
10 x 640-MW units were installed 

24 billion KWt / year = 10% of Siberian need 
December 1978 - first unit in operation 
December 1985 - 10th unit in operation 

2000 - plant officially commissioned 

2007 - 2011: 2" plan of the plant 
modernization / refurbishment 

2005 - 2010: Massive replacement of control 
and protection systems on all units / 
installation of DCS 


SEQUENCE 
OF THE ACCIDENT 












Fire at Other Plant started sequence 
of the events resulted in the accident 


There was fire in a communication 
room at Bratskaya hydro 









Communication with Dispatch There was order to use Sayano 
Center was interrupted for 30 min for the grid regulation 


Several Invalid Assumptions 
During Operation Of The Plant 


Unit 2 was selected as the leading 
Ё —,—— _ | to regulate the grid, as the most 
= “reliable” (recently after maintenance) 
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The Grid was regulated only by 
Sayano plant — more pronounced 
Change of parameters 
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Units 1,2,4,5,7 and 9 controlled changes | 


Units 8 & 10 were with base load 
of MW demand & frequency of the grid Unit 6 was in stand-by 


Before accident Unit 2 six times was in non- 
recommended Zone swinging from 170 to 
600-MW 
















Grid regulation was 
accompanied by very 
high vibration at Unit 2 


New vibration control of Unit 2 | 
was installed in 2009 
but it was out of service 
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in the control room 


The Bolts and Vibration at Unit 2 










13 min before accident 
the limit of vibration was 
exceeded 3.75 times 







At the accident 
the limit of vibration was 
exceeded 5.25 times 







Turbine Cover Bolts Failed on Unit 2 





High vibration contributed to 
- | the bolts fatigue, their functional capacity 
was lost, the turbine cover was opened. 





Turbine Cover Bolts Failed on Unit 2 


High pressure on large surface had 
created in the unit pit enormous 
uplifting force 





Unit 2 Was Brutally Lifted 


Количество воды, поступающей E напорный 
водовод, можно регулировать верх ней 
заслонной. Пустой водовод следует 
заполнять постепенно 





В турбину вода попадает танке через 
регули руемую заслон ну с высоты 170 м 


Если Ha нападвижнуұю тұрбину сразу подать 
воду пол мансимальным напором, она 


жне пропустит» ве через себя. По закону 
сообщающихся co судов потон рванет вверх 




















The unit weight is 2,691-t, the rotor weight is 900-1 





Flooding of the Powerhouse Started 








Flooding of Transformers 





Section of the Powerhouse 
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Short Circuits Caused by Water 





- 
ooo 


Sudden full flooding of the powerhouse disabled 
controls and protections of the units; 
The control systems stopped operating (no normal 
and emergency electricity supply) 
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Overtopping Exposure in 2 days 


All spillway gates were opened 
allel 2-day effort: from 11:32 am 


Electric generator was 
borrowed to open 
the spillway eles 





Apparently tie reservoir mm m 
overtopping exposure in 2 days after the accident, 
if the spillway would not be opened completely 











Some Issues During the Accident 
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Some Issues During the Accident 


Access door to controls of the unit 2 


emergency gate was closed Pc ES „| apocket flashlight was used 
— forced entrance was required Kolb. о EE ee 





Emergency Plan Did Not Exist 


News apparently were 3 hours 1 Emergency situation lasted 1-hr 7-min, 
after the accident Safety Manager abandoned the plant 


HN) (ger О ешеш Lack of emergency procedures 
for Emergency situation 








Evacuation Of People 


People were not oriented what to do; only oral orders were 
contemplated in a case of emergency 





Emergency exit signs did 
not exist to direct 
people to safety places 









No drills to evaluate 
preparedness in the past 


Emergency Electricity Supply Does Not Function 
— Manual Attempt To Open The Gate 





The Gate Area Several Minutes Later 
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LOSS OF HUMAN LIVES 
AND SCOPE OF DAMAGE 








75 Persons Died 


о All persons who were inside the 
powerhouse at elevation 335-m a.s.l. 
and below have been perished. 


o 10 persons from the plant and 65 
maintenance contractors died. 

o There were app. 300 persons at the 
plant at 8:13 a.m. (at the time of 
accident). 


o Normal plant shift is app. 12 persons. 


Powerhouse Before & After Accident 


Total loss of equipment Re-build time 5 years, 
inside the powerhouse Costs over 1.3 billion USD 
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CBI / Contingency Business 
Interruption losses for 
aluminum smelters 


Shutdown will push up 
market prices in Siberia’s grid 





Total Destruction of Unit # 2 





Generators 7 And 9 Destroyed 
By Short Circuits 










dc P 


D 


“ey N E » te 
—: # » L uet 


Destroyed generator crosshead 
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Area Of The Disappeared Section Of 
The Powerhouse 





Major Losses 


Water under high pressure 
increased the initial damage 


Severe damage to main concrete structure 
of the powerhouse and 
partial collapse of the roof and walls. 
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Major Losses (cont.) 


Concrete elements were destroyed by brutal 
destruction caused by elevation of unit 2, high 
pressure jet streams and collapse of the structure 














Damaged Transformers 








Environmental Impact 





The spill flowed 
along the river 
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100-t of oil had been 
inda into the river 1 
] —| Environmental scandal 
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WHY IT HAPPENED? 


TECHNICAL CAUSES / 
HARDWARE 





Unit 2 Should Be Shutdown In April 
Or May 2009 


Изменение показаний датчика радиальных вибраций ТПНБ подшипника 
турбины при мощностях 500-600 МВт 
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Vibration trend before the accident 


Causes Of The Unit 2 Failure 








Poor 2009 maintenance 
- bolts fatigue was 
not corrected 


New 2009 vibration system 
was out operation — 
No response by control room 


Causes Of The Unit 2 Failure (Cont.) 












Numerous power swings 
being only Grid regulation 
- lasted high vibration 
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ew 2008 desig 
| had structural deficiencies — excessive 
time of operation in Zone 2 


Aggravating Factors Before The 
Accident 


о Prototype character of the Grid regulator (operational test 
in 2008; commissioned on 21.07.2009) 


о No vibration trip 


o 212-m reservoir level above designed 197-m made 
operation longer in Zone 2 of high vibration. 


o Lack of criteria to operate / vibration and strange sounds 
were noted long time before 


О Design of the bolts - no maintenance requirements, no 
forelock on the nuts 


O New controls of the wicket gate at Units 2,5 & 6 from 
2009 (prototype?) 


o — Worn out surfaces in the bearings including the shaft 
contributed to higher vibration 


O Cavitation had contributed to vibration / unbalanced rotor 


о Unit 2 was at the end of its useful life (29 years апа 9 
months vs. specified 30 years) 


о Useful life of the failed turbine bolts was also 30 years. 











Fatigue Appeared And Developed 
When Units Worked In Zone 2 
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All Plant Units Operated In Zone 2 
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In 2009 unit 2 was 
232 times in zone 2 
totaling 46 minutes 
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In 2009 unit 4 was 
490 times in zone 2 
totaling 1-hr 38 minutes 


Was The Grid Regulator "GRARM" 
Forcing Units To Work In Zone 2? 





o OEM had not approved algorithm of the grid 
regulator 

o Criteria for work priorities in the group 
regulation were not established 

o  Nocriteria for selection of the leading unit 
neither how long to maintain it regulating 

o Specific design and the camp of operational 
characteristics of the units were not considered 

o Individual power limits and specific physical 
conditions and vibration behavior of each unit in 
zone 2 were not considered 


High Vibration Was From 1982 









Vibration after maintenance 
should be 38% of the limit | | 
| Vibration after 2009 maintenance 
| | was 93% of the limit 


Pit, Turbine Cover and the 


Turbine 
Bolts 
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At Least 6 Nuts Were Not Installed, 
Total 80 Bolts X 80-mm Diameter 


The nuts were 
not forelocked by design 





OEM did not specify 
requirements for the bolts maintenance 
and inspections: result-badly worn bolts 








47 Bolts Of 49 Laboratory Tested 
Were Defective (All From Unit 2) 





Picture is shown for reference only as example 





Issues With Design Of The Units 


Issues With Design - Conditions Of 
The Turbine Wheels 


o The turbine wheels had to be repaired every 9k 
to 10,000 hours because cracking of the blades 
is permanent. 


o Use of welding was required to repair cracks up 
to 130-mm and cavitation loss of metal up to 12- 
mm deep. 


o More repairs mean higher exposure to 
breakdown. There should be limits how many 
times the turbine wheels may be repaired. 


o Recommendation to replace the worn out wheels 
was never implemented. 





WHY IT HAPPENED? 


HUMAN ELEMENTS / 
SOFTWARE 





Aggravation Factors Of The 
Accident 


i. Errors in design of the plant and 





equipment 

2. Lack of investment to replace obsolete 
equipment 

3. Poor maintenance and operational 
standards 


4. Qross negligence and carelessness of 
management at all levels 





Design Errors 


O 


Plant had no facility to close from the control room the 
emergency gates at the penstocks (no manually 
operated button). 


Emergency gates at the penstocks did not close when 
electricity supply fails (in case of over speed-yes). 


Wicket gates did not close when electricity supply fails. 


There was no separate and totally independent back-up 
electricity supply system installed. 


Architecture of the control systems was not uniform for 
all units. 


No trip on high vibration 


Protection devices and circuitry was not dust- and 
water-proofed 


Design and Operational Issues 


o Significant increase of scope of maintenance 
work was noted after 50,000-hr of unit operation. 
As consequence more people were required or 
not all work done. 


o New instruction to evaluate the risk of operation 
has canceled a number of previous documents 
related to safety standards. That was against a 
general trend that equipment was getting more 
obsolete and deteriorated. 

o Cost-cutting on safety: Safety standards had 
been simplified and several safety documents 
are not anymore in use from 2006. 





Poor Conditions Of The Plant Were 
Well Known 


о Ex-General Manager denounced the 
critical condition of the plant many years 
ago. 

o 2007 survey of the Russian Accounting 
Chamber: use of the plant is dangerous, 
the prosecutor general's office was 
approached 

o 2007 info warned: 85% of the equipment 
was obsolete; most of the equipment was 
worn-out. 





Poor Conditions Of The Plant Were 
Well Known (cont. ) 


o Before 2000 Rostechnadzor (jurisdictional body) had 
stated that the operation was unsafe 


o 2007 and 2008 surveys done by Rostechnadzor 
could not prevent the accident (no issues found — 
suspected corruption) 

o Plant was commissioned officially only in 2000 
because of problems to finish project but commercial 
operation started in 1978 

o 2000 reception of the plant was based on 1989-1991 
documentation: real conditions were not documented 

o General quality of the project and equipment was 
declared in 2000 as “GOOD” 





It Was A Stockholders Business 
To Maintain The Plant In Safety Conditions 


(reply of the prosecutor general’s office in 2007) 


Jurisdiction surveys responsible 
to supervise safety were 
misleading due to corruption 





THREE MAJOR ISSUES WITH 
THE DAM AND SPILLWAY 





Major Issues of the Dam 









Cause of cracks in the dam: 
| operation of the dam started when 












By design the dissipater pit |" 1 
was not able to resist \ 


Weaken inter-phase dam-rock; 
Filtrations greater 
than estimated by design 










Deterioration Of The Dam 
Had Started During Construction 


СЕНЕ 
->| Preparation for construction 
а continued 12 years 

y instead of 5 years by design 


ET: \ \ 
і! + t 


za Political optimism of the epoch put pressure to short time 

| of construction phases and save on concrete volume 
thus the arch-gravity design was selected 

(4096 arch load and 6096 gravity) 


Dam Was Repaired In 1996 & 1998 


Cracks in the dam were repaired Dozens of thousands logs 
in 1998 with epoxy materials may be present in reservoir 


Restoration of injected impermeable № Restoration of injected impermeable | 
Screen on the right bank ro Screen Under the dam 








The Energy Dissipater Requires 
Repair Now! 


Dissipater pit had been 
destroyed in 1986 and 1998 










In 1986 75% of concrete 
underlying blocks and 
30,000-m3 of concrete 

were destroyed 





Issue resulted in rock 
damage and redistribution 
of rock stresses 
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Solution of the Spillway Issue 





Additional lateral spillway is 
under construction at the present time 





The Plant Had Been Flooded 
Twice In The Past : in 1979 and 1985 
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ISSUES WITH ELECTRICITY 
SECTOR 





o In Russia 
o Inthe company 
o At the power plant 










Some Issues in Russian Electricity 
Sector 








46 regional managers of 84 in 
Rostechnadzor (jurisdictional 
body) were fired recently 
(presumably corruption) 








Efficient State Centralized 
Supervision for Hydros 
had disappeared from 2000 
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Unclear rules of privatization 
interrupted normal rate 

of investment in maintenance 

and replacement of technology 


Blame-on environment 
at state, company and plant level 












Collapse of USSR resulted in collapse 
of cooperation and partnership 
with issues of supply and quality 
of the products and service 










Some Issues in Russian Electricity 
Sector (cont.) 






Major accent on making money 
then on sound technical policy; 
this is known in the Russian press 
as “Factor of Successful Manager” 


Decreased efficiency of 
communication inside 
the companies & with contractors 











Sei HE 





mansk 
ғ 


Kara Sea 







Increasing lack of 
qualified labor for 
operation and maintenance 






Interrupted continuity of 
operational standards between 
old & new generation of specialists 
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Issues at the Plant 


o Extended life of some elements beyond 
recommendations of OEM (i.e. control systems) 

o OEM/LMZ was excluded to provide maintenance to 
the plant (local companies were benefited) 

o Careless and greedy plant management (conclusions 
of the commission) who was shareholders of the 
contracted local companies. 

o Manufacturer of the turbine bolts that failed was not 
invited to diagnostic their conditions (Unit 2 from 1979) 

o Operational and maintenance requirements were not 
understood and clearly expressed 








Negligence At The Plant 


O 


Criminal behavior of personnel that failed to 
recognize danger — formal accusations had been 
presented to the court. 


Fatigue cracks in the bolts had been reported but 
corrective measures were not taken. No NDT 
applied for evaluation. 

Maintenance contracts did not specify 
requirements for quality control of works. 

Failure to comply with technical instructions. 
During maintenance works some defects were 
not repaired. 


CONCLUSIONS 





Specific points to be discussed during 
Power Gen engineering surveys in Russia / 


From insurance point of view 


Consequences Of The Accident 


о Total loss of the powerhouse is now a real 
loss scenario for large plants (before total 
flooding of the powerhouse was related 
probably with a dam break). 


о Cost of human error may be significantly 
higher for large plants then for small ones. 

o Lack of the loss control programs and 
emergency plans is significant deficiency. 

o Permanent drills are required to know real 
preparedness. 








Background of the Russian 
Generation Industry 


O 


O 


O 


Obsolete critical equipment is in use beyond the 
useful life specified by OEM - higher that normal 
Machinery Breakdown (MB) exposure. 


Vulnerability associated with quality of 
modernization including gross negligence. 


Prototype character of the critical items during 
modernization of large power plants - issues with 
quality of supplied items increase MB exposure. 


Impact of poor technological discipline 
(misleading certificates and misrepresented 
evaluations, quality control, qualification of labor 
and other issues). 





Specific Check Points For The Bolts 


O 


O 


Of The Hydraulic Turbine Covers 


Useful time of life, scope of maintenance 
and NDT inspections of the bolts. 


Operational instructions and limits to 
work in the non-recommended zones of 
the unit to avoid initiation of the fatigue 
cracking (swing of power, vibrations 
limits). 

Installed system of the Grid regulation: 
behavior in the non-recommended 
zones, manufacturer, when installed and 
upgraded, OEM approval. 


Specific Check Points For The Bolts 
Of The Hydraulic Turbine Covers (cont. 


o Control of number of the operations and time of 
operation inside the non-recommended zone to 
adjust frequency of inspections and maintenance 


o Operational vibration system with capacity for 
vibration analysis (vibration spectrum, recording 
of the data etc.) 

o How the grid regulation facility takes into 
consideration real physical conditions of the unit 
(levels of vibration, hydrodynamic unbalance 
in the turbine, conditions of the turbine wheel / 
cracks, number of repairs and vulnerability to 
cracking, loss of metal by cavitation and erosion, 
conditions of bearings and other factors) 





Specific Check Points For The Bolts 
Of The Hydraulic Turbine Covers (cont) 


o Independent emergency generator for activation 
of the spillway gates and units’ emergency gates 
at water conducts. 

o Independent emergency generator for the 
powerhouse in addition to battery supported 
systems 

o Emergency closing (trip) of the guide vanes / 
wicket gates and the units emergency gates 
when electricity supply failed 

o Remote manual closing of the units emergency 
gates from the control room (stop water flow) 

o Other important points not mentioned here. 





Specific Check Points For the Plant 
Equipment 


o List of equipment and critical elements with 
expired useful life defined by OEM specification in 
order to control obsolescence and extreme 
degradation 

o What is percentage of the obsolete equipment 
that the plant has at the present time? 

o Do all obsolete equipment included into the 
modernization programs? 

o What are prototype solutions during upgrading, 
refurbishing and modernization (whose failure 
may result in major loss)? 





Specific Check Points For the Plant 
Equipment (cont.) 


o Dynamics and execution of the modernization 
don / percentage of investment, availability 
of the budgeted funds 


o Spill capacity of the dam versus updated flood 
flows for specific return periods (recommended 
update every 10 years). Emergency watershed 
regulation should be in use when there is an 
issue with spill capacity - this is in function of 
social risks downstream the dam. 


o Conditions of Interfaces dam-rock: injection 
screens that control filtrations under the dam 
and through the dam abutments on both banks. 
Alarm when filtrations are higher than the design 
parameters. 








Specific Check Points For the Plant 
Equipment (cont.) 


o Identified the design errors and mitigation 
actions; what was done? 

o History to respond to the Critical items that were 
on the punch-list and quality of solutions. 

o List of the Critical spare parts in warehouses and 
their supply from OEM (collapse of previous 
scheme of cooperation may result in long time of 
supply and non-existence of providers). 

o What is useful life of the turbine wheel and what 
is its vulnerability in function of the number of 
the turbine wheel repairs? 





Specific Check Points For the Plant 
Equipment (cont.) 


o Normal and emergency electric supply systems 
separated at all levels including cable routing. 


o Preferable dust- and water-proof execution of the 
elements, devices, panels and cabling related 
with the plant critical protection. 

o Cable routing and potential impact of fires on 
reliability of the protection systems. 

o Vulnerability of the plant when the units have 
different architecture of the control systems, as 
consequence of modernization. 





Specific Check Points For the Plant 
Equipment (cont.) 


o Where turbines (hydro, steam or 
gas) have no automatic vibration 
trips then there should always be 
clear instructions on the action to be 
taken and at what vibration level. 


o The operator should always have full 
trip authority when the vibration 
reaches the clearly identified 'advise 


trip' level. 


Human Elements / Software 


o What are the documents to evaluate the risk of 
the plant operation; when there were modified? 
Were they simplified at some point? 

o Modification of the safety standards: were they 
simplified? date of the last revision. 

o Desk top simulations of the emergency situations 
to prevent the team failure to make adequate 
critical decisions. 

o Quality of the Emergency plans: major loss 
scenarios covered, instructions in written, 
personnel trained, external help, etc. 

o Quality of drills. 





Human Elements / Software (cont.) 


o Emergency training of the personnel 
based on identification of all critical loss 
scenarios. 


o Impairment controls applied for 
protection systems to assure permanent 
protection. 

o Maintenance standards and how they are 
controlled. 

o How maintenance contracts specify 
s a rid of the Quality Control and 

ow it is supervised? 





Human Elements / Software (cont.) 


o Latest jurisdictional documents of Rostechnadzor 
(note: in the past the conclusions were 
misleading and misrepresented). 

o Supervision of overhauls by OEM. 

o Quality of the new service providers and 
equipment suppliers / screening. 

o Qualification and certification of labor at the 
present time: educational background, training, 
experience, turn over. 

o Quality of LOTO: logout / tagout procedures; 
closed doors to critical equipment and 
accessibility to it in case of emergency, etc. 





Human Elements / Software (cont.) 


o Escape routes clearly marked, 
emergency lighting available 
(autonomous with own batteries and 
connected to the emergency supply 
circuit), portable flashlights available. 


o Vulnerability of communication with the 
dispatch center, watershed regulating 
еч for floods control, external 

elp. 

o Existence of Blame-on environment. 

o Other missed points. 





